1. Field of the Invention
This invention is related to the general field of academic assessment testing, and within that field to the task of determining test-taker classifications relative to performance criteria in tests which contain at least one open-ended item. Although the methods described herein may be useful in other related tasks, the most common application involves tests for which performance standards are established and the most significant outcome of the test is the classification of test-takers into two or more performance levels.
2. Performance Standards
The current invention's primary application is to tests which are designed to measure whether or not test-takers have achieved specific performance standards.
Performance standards are used to determine whether or not the test-taker has demonstrated a required level of performance or achievement, or to classify the test-taker in terms of two or more “performance levels”. For any given test, each performance level is represented as a range of scores on a test. The division between one performance level and an adjacent performance level is defined by a single score on the test, and such a score can be referred to as a performance cutoff score, or cutoff score. Test-takers who score at or above the cutoff score are deemed to have demonstrated the required level of achievement or proficiency represented by the standard for the upper performance level, while test-takers who score below the cutoff score are deemed to have not demonstrated the required level and are classified in the lower performance level. (In some instances the cutoff score is specified as the highest score below the standard rather than as the first score meeting or exceeding the standard.)
In K-12 tests, such standards are commonly used in “barrier” tests on which the student must meet the standard to continue on a normal academic path. In many state-wide tests, failure to meet the standard results in significant consequences including (a) being denied graduation, (b) being allowed to graduate but without a full diploma, (c) not being allowed to go on to the next grade, and (d) having to go to summer school in order to be promoted to the next grade.
In typical professional examinations, the applicant must meet the standard in order to achieve certification and therefore must reach or exceed the cutoff score in order to become certified or approved.
For some tests, multiple performance levels are established with cutoff scores separating each successive level.
Open-Ended Items
The current invention is further focused on tests that contain one or more open-ended items which are included in determining the test-taker's score relative to the cutoff score. Such tests may or may not include multiple-choice and other objective assessment items in addition to open-ended items. Such multiple-choice and other objective assessment items have proven to be cost effective, and there are common practices in the educational assessment industry to ensure accurate scoring for such multiple-choice items. In addition, to further enhance the precision of scores of multiple choice items in situations where a precise score is critical to the certainty of the performance level determination, it has been proposed in the administration of state-wide tests to automatically re-score the test of any student whose first score is within a specified range below a cutoff score.
For a number of reasons, over the past decade, open-ended items have been increasingly adopted, especially in state-wide tests, and such items are not scored with the precision of scoring multiple-choice and other objective assessment items.
Typically such open-ended items are scored by human “readers”; that is, each response is read by one or more trained readers who assign appropriate scores to the response.
Reliability of Reader Scores
It is necessary to train readers so that all readers will give equivalent scores to assessment responses, and there are well established procedures in the industry to provide such training and to test readers for accuracy before they are allowed to read and score test-taker responses. Nonetheless, readers do not give identical scores to test-taker responses. While a few readers will be close to accurate most the time, many readers will show patterns of inaccuracies. Such patterns include being overly harsh, overly lenient, and being inconsistent. In some instances, readers can be characterized by the patterns of their inconsistencies. In all instances, readers can be characterized in terms of their accuracy in assigning scores to open-ended items.
Reader accuracy can be modeled using what is known as the Hierarchical Rater Model, “HRM” (Richard J. Patz, Markov Chain Monte Carlo Methods For Item Response Theory Models With Applications For NAEP, (1996)(unpublished Ph.D. dissertation, Carnegie Mellon University); Brian W. Junker and Richard J. Patz, The Hierarchical Rater Model For Rated Test Items, proceedings of the Psychometric Society, Champaign-Urbana, Ill., USA, June, 1998; Richard J. Patz, et al, The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment Data, March 1999; Richard J. Patz, et al., The Hierarchical Rater Model for Rated Test Items and its Application to Large-Scale Educational Assessment Data, March 2002), although other models of rater accuracy and other procedures can be employed. Reader (or rater) accuracy is typically estimated by data from one or more of three common processes: monitoring items, lead reader review, and inter-reader agreement.
Monitoring items: Monitoring items are test-taker responses for which the correct scores have been determined and vetted by senior project personnel. Such items can also be called “validity items” or “check set” items. As part of the scoring process, such monitoring items are intermixed with unscored test-taker responses for assignment of scores by the reader. The extent to which a reader's scores match the correct scores indicates the accuracy of the reader.
Lead reader review: In lead reader review, the scores assigned by the reader are checked by a skilled “lead” reader. In this procedure, the lead reader scores a subset of the items scored by the reader and checks the score assigned by the reader for accuracy. The extent to which the lead reader identifies the reader scores as accurate indicates the accuracy of the reader.
Inter-reader agreement: In scoring some items such as essays and other items requiring extended responses, each test-taker response is often scored independently by two different readers. While standard practices use two independent readers, any number of readers can independently score an item. The extent to which readers'scores match is an indication of inter-reader agreement and is a measure of consistency in scoring across readers. For each reader, the extent to which his or her scores match those of other readers can be considered a measure of the reader's accuracy, or at least a measure of the reader's agreement with other readers in the project.
Determining the “Correct Score” for Open-Ended Items
Because readers are not 100% accurate, the score given by a reader can not be assumed to be the correct score, but only a single estimate or measurement of the test-taker's score on the assessment item.
By using measures to determine reader accuracy, such as those listed above, the extent to which test-taker scores may be inaccurate can be estimated. However, measures of reader accuracy do not directly offer any opportunity to correct errors in scores, but merely describe the extent to which reader assigned scores may be considered fair representations of the “correct scores” for the assessment items.
Improving the Precision of Test-Taker Scores While Determining Reader Aaccuracy:
Although the determination of reader accuracy does not directly offer a mechanism to adjust or correct test-taker scores, the processes shown above to determine reader accuracy, offer opportunities to increase the accuracy of scores given to test-takers.
Monitoring items: If a reader is deemed to be incorrectly scoring items, scores from the reader may be removed from a set of items and those items can be scored by another reader as if the items had never been read. In such instances, the test-takers whose items are read by another reader may or may not end up with scores that are more accurate than the scores originally given. In general, however, monitoring items are used to measure reader accuracy and do not cause test-taker scores to be altered.
Lead reader review: As with monitoring items, lead reader review is primarily directed to measure reader accuracy and not alter reader scores. However, it is possible to override the score assigned by a reader with the score assigned by the lead reader or with a score that is a fiction of both the reader and the lead reader. While such a procedure would seem to improve the accuracy of test-taker scores, it suffers from two systemic problems. First, even if a lead reader scores are substituted for a reader score, the resulting test-taker score is still based on a single reader's judgement. Secondly, only a small percentage of test-taker scores are read by lead readers, usually around 10 percent, so that approximately 90% of the test-takers have no opportunity to receive a more reliable score than that originally given.
Inter-reader agreement: Unlike the above two procedures, having two independent readers read the same item is designed primarily to increase the accuracy of test-taker scores. In statistics, it has long been known that measurement accuracy will increase with more independent measures of the same thing. Thus, scores based on the average (or sum) of two independent readers will be more accurate than one reader, and precision will increase as the number of independent reads of a given item increases.
In addition, in general industry practice, the use of two readers is further refined to provide “resolution”, or correction, when the two readers disagree. In standard practice, pairs of scores from two independent readers are characterized as “discrepant” or “not discrepant”. In all cases, exactly the same scores from two independent readers are “not discrepant”. Depending on the item being scored, adjacent scores may be characterized as either discrepant or not discrepant. In almost all cases, scores with an absolute difference more than one are considered discrepant. If scores from two readers are “discrepant”, a third read is performed, often from a highly knowledgeable and reliable reader or administrator. While the manner in which this third score is utilized varies, the test-taker is increasingly likely to have a “correct” score when two readers agree or when a third reader has adjudicated the disagreement between the two readers; at least relative to having only a single reader score the item.
Accordingly presently employed assessment scoring schemes do not incorporate a determination of the likely result of the allocation of scoring resources, and thus, limited scoring resources are often used inefficiently. Scoring resources are often used in situations in which the likely outcome of the allocation of scoring resources is that the test-taker's performance level will not change. Conversely, situations may arise in which scoring resources are not allocated in a circumstance in which there is a significant likelihood that the outcome of the allocation of scoring resources would have changed the test-taker's performance level. Thus, under presently used assessment scoring schemes, scoring resources are overused in some circumstances and underused in other circumstances.